Q-functionals for Value-Based Continuous Control

We present Q-functionals, an alternative architecture for continuous control deep reinforcement learning. Instead of returning a single value state-action pair, our network transforms state into function that can be rapidly evaluated in parallel many actions, allowing us to efficiently choose high-value actions through sampling. This contrasts with the typical off-policy control, where policy is trained sole purpose selecting from Q-function. represent action-dependent Q-function as weighted sum basis functions (Fourier, Polynomial, etc) over action space, weights are state-dependent and output by Q-functional network. Fast sampling makes practical variety techniques require Monte-Carlo integration Q-functions, enables action-selection strategies besides simple value-maximization. characterize framework, describe various implementations demonstrate strong performance on suite tasks.